System and method for use of crowdsourced microphone or other information with a digital media content environment

ABSTRACT

In accordance with an embodiment, described herein is a system and method for use of crowdsourced microphone or other information with a digital media content environment. A user may bring their media device with media application to a venue/performance, for example a theater, or a music concert, or other event which the user desires to attend. A microphone and/or other hardware components of the media device can be used to indicate a presence of the media device. The system can also, or alternatively, utilize an interaction information by the media device or the user thereof with other systems or environments. The media application can be configured so that, upon determining that the media device is present within a particular venue or attending a particular performance, the media server can enable one or more venue-aware media application features, to provide various types of functionalities that improve the user experience.

CLAIM OF PRIORITY

This application claims the benefit of priority to U.S. ProvisionalPatent Application titled “SYSTEM AND METHOD FOR USE OF CROWDSOURCEDMICROPHONE OR OTHER INFORMATION WITH A DIGITAL MEDIA CONTENTENVIRONMENT”, Application No. 62/440,932, filed Dec. 30, 2016; whichapplication is herein incorporated by reference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

Embodiments of the invention are generally related to providing digitalmedia content, and are particularly related to a system and method foruse of crowdsourced microphone or other information with a digital mediacontent environment.

BACKGROUND

Today's consumers enjoy the ability to access a tremendous amount ofmedia content, such as music and videos, using a wide variety of mediadevices. Digital media content environments, for example media streamingservices such as Spotify, enable users to stream music content thatcomplements their lifestyle, while commuting, exercising, relaxing athome, or engaging in other types of activities. Such digital mediacontent environments are well suited to offering additionallifestyle-oriented functionality that can provide an improved userexperience. However, traditional digital media technologies are limitedin the extent to which they can integrate with the user's physicalenvironment.

SUMMARY

In accordance with an embodiment, described herein is a system andmethod for use of crowdsourced microphone or other information with adigital media content environment. A user may bring their media devicewith media application to a venue/performance, for example a theater, ora music concert, or other event which the user desires to attend. Amicrophone and/or other hardware components of the media device can beused to indicate a presence of the media device. The system can also, oralternatively, utilize an interaction information by the media device orthe user thereof with other systems or environments. The mediaapplication can be configured so that, upon determining that the mediadevice is present within a particular venue or attending a particularperformance, the media server can enable one or more venue-aware mediaapplication features, to provide various types of functionalities thatimprove the user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example digital media content environment, inaccordance with an embodiment.

FIG. 2 illustrates use of crowdsourced microphone or other informationwith a digital media content environment, in accordance with anembodiment.

FIG. 3 further illustrates use of crowdsourced microphone or otherinformation with a digital media content environment, in accordance withan embodiment.

FIG. 4 illustrates a flowchart of a method for use of crowdsourcedmicrophone or other information with a digital media contentenvironment, in accordance with an embodiment.

DETAILED DESCRIPTION

As described above, digital media content environments, for examplemedia streaming services such as Spotify, enable users to stream musiccontent that complements their lifestyle, while commuting, exercising,relaxing at home, or engaging in other types of activities; and are wellsuited to offering additional lifestyle-oriented functionality that canprovide an improved user experience. However, traditional digital mediatechnologies are limited in the extent to which they can integrate withthe user's physical environment.

In accordance with an embodiment, described herein is a system andmethod for use of crowdsourced microphone or other information with adigital media content environment. A user may bring their media devicewith media application to a venue/performance, for example a theater, ora music concert, or other event which the user desires to attend. Amicrophone and/or other hardware components of the media device can beused to indicate a presence of the media device. The system can also, oralternatively, utilize an interaction information by the media device orthe user thereof with other systems or environments. The mediaapplication can be configured so that, upon determining that the mediadevice is present within a particular venue or attending a particularperformance, the media server can enable one or more venue-aware mediaapplication features, to provide various types of functionalities thatimprove the user experience.

Digital Media Content Environments

FIG. 1 illustrates an example digital media content environment, inaccordance with an embodiment.

As illustrated in FIG. 1, in accordance with an embodiment, a mediadevice 102, operating as a client device, can receive and play mediacontent provided by a media server system 142 (media server), or byanother system or peer device. In accordance with an embodiment, themedia device can be, for example, a personal computer system, handheldentertainment device, tablet device, smartphone, television, audiospeaker, in-car entertainment system, or other type of electronic ormedia device that is adapted or able to prepare a media content forpresentation, control the presentation of media content, and/or play orotherwise present media content.

In accordance with an embodiment, each of the media device and the mediaserver can include, respectively, one or more physical device orcomputer hardware resources 104, 144, such as one or more processors(CPU), physical memory, network components, or other types of hardwareresources.

Although, for purposes of illustration, a single client media device andmedia server are shown, in accordance with an embodiment a media servercan support the simultaneous use of a plurality of client media devices.Similarly, in accordance with an embodiment, a client media device canaccess media content provided by a plurality of media servers, or switchbetween different media streams produced by one or more media servers.

In accordance with an embodiment, the media device can optionallyinclude a touch-enabled or other type of display screen having a userinterface 106, which is adapted to display media options, for example asan array of media tiles, thumbnails, or other format, and to determine auser interaction or input. Selecting a particular media option, forexample a particular media tile or thumbnail, can be used as a commandby a user and/or the media device, to the media server, to download,stream or otherwise access a corresponding particular media content itemor stream of media content.

In accordance with an embodiment, the media device can also include asoftware media application 108, together with an in-memory client-sidemedia content buffer 110, and a client-side data buffering logic orsoftware component 112, which can be provided as software or programcode that is executable by a computer system or other processing device,and which can be used to control the playback of media content receivedfrom the media server, for playing either at a requesting media device(i.e., controlling device) or at a controlled media device (i.e.,controlled device), in the manner of a remote control.

In accordance with an embodiment, a connected media environment logic orsoftware component 120, which can be provided as software or programcode that is executable by a computer system or other processing device,can be provided at the media device, either as part of the mediaapplication, or separately, for example as a firmware, to enable themedia device to participate within a connected media environment (e.g.,a Spotify Connect environment) that enables a user to control theplayback of media content at such controlled devices.

In accordance with an embodiment, the client-side data buffering logic,together with the media content buffer, enables a portion of mediacontent items, or samples thereof, to be pre-buffered at a client mediadevice. For example, while media options are being prepared for displayon a user interface, e.g., as media tiles or thumbnails, their relatedmedia content can be pre-buffered at the same time, and cached by one ormore media devices in their media content buffers, for prompt andefficient playback when required.

In accordance with an embodiment, the media server can include anoperating system 146 or other processing environment which supportsexecution of a media server 150 that can be used, for example, to streammusic, video, or other forms of media content to a client media device,or to a controlled device.

In accordance with an embodiment, the media server can provide asubscription-based media streaming service, for which a client mediadevice or user can have an associated account and credentials, and whichenable the user's media device to communicate with and receive contentfrom the media server. A received media-access request from a clientmedia device can include information such as, for example, a networkaddress, which identifies a destination media device to which the mediaserver should stream or otherwise provide media content, in response toprocessing the media-access request.

For example, a user may own several media devices, such as a smartphoneand an audio speaker, which can play media content received from a mediaserver. In accordance with an embodiment, identifying informationprovided with a media-access request can include an identifier, such asan IP address, MAC address, or device name, which identifies that themedia-access request is intended for use with a particular destinationdevice. This allows a user, for example, to use their smartphone as acontrolling device, and their audio speaker as a controlled device towhich media content should be sent. The media server can then send therequested media and/or forward the media-access request to the audiospeaker, even though the request originated at the user's smartphone.

In accordance with an embodiment, one or more application interface(s)148 can receive requests from client media devices, or from othersystems, to retrieve media content from the media server. A contextdatabase 162 can store data associated with the presentation of mediacontent by a client media device, including, for example, a currentposition within a media stream that is being presented by the mediadevice, or a playlist associated with the media stream, or one or morepreviously-indicated user playback preferences. The media server cantransmit context information associated with a media stream to a mediadevice that is presenting that stream, so that the context informationcan be used by the device, and/or displayed to the user. The contextdatabase can be used to store a media device's current media state atthe media server, and synchronize that state between devices, in acloud-like manner. Alternatively, media state can be shared in apeer-to-peer manner, wherein each device is aware of its own currentmedia state which is then synchronized with other devices as needed.

For example, in accordance with an embodiment, when the destinationmedia device to which the media content is being streamed changes, sayfrom a controlling device to a controlled device, or from a firstcontrolled device to a second controlled device, then the media servercan transmit context information associated with an active media contentto the newly-appointed destination device, for use by that device inplaying the media content.

In accordance with an embodiment, a media content database 164 caninclude media content, for example music, songs, videos, movies, orother media content, together with metadata describing that mediacontent. The metadata can be used to enable users and client mediadevices to search within repositories of media content, to locateparticular media content items.

In accordance with an embodiment, a server-side media content bufferinglogic or software component 180, which can be provided as software orprogram code that is executable by a computer system or other processingdevice, can be used to retrieve or otherwise access media content items,in response to requests from client media devices or other systems, andto populate a server-side media content buffer 181, at a media deliverycomponent or streaming service 152, which can be similarly provided assoftware or program code that is executable by a computer system orother processing device, with streams 182, 184, 186 of correspondingmedia content data, which can then be returned to the requesting deviceor to a controlled device.

In accordance with an embodiment, a plurality of client media devices,media server systems, and/or controlled devices, can communicate withone another using a network, for example the Internet 190, a local areanetwork, peer-to-peer connection, wireless or cellular network, or otherform of network. For example, a user 192 can interact 194 with the userinterface at a client media device, and issue requests to access mediacontent, for example the playing of a selected music or video item attheir device, or at a controlled device, or the streaming of a mediachannel or video stream to their device, or to a controlled device.

In accordance with an embodiment, the user's selection of a particularmedia option can be communicated 196 to the media server, via theserver's application interface. The media server can populate itsserver-side media content buffer at the server 204, with correspondingmedia content, 206 including one or more streams of media content data,and can then communicate 208 the selected media content to the user'smedia device, or to a controlled device as appropriate, where it can bebuffered in a client-side media content buffer for playing at thedevice.

In accordance with an embodiment, and as further described below, thesystem can include a server-side media gateway or access point 220,which can be provided as software or program code that is executable bya computer system or other processing device, or other process orcomponent, which operates as a load balancer in providing access to oneor more servers, for use in processing requests at those servers. Thesystem can enable communication between a client media device and aserver, via an access point at the server, and optionally the use of oneor more routers, to allow requests from the client media device to beprocessed either at that server and/or at other servers.

For example, in a Spotify media content environment, Spotify clientsoperating on media devices can connect to various Spotify back-endprocesses via a Spotify “accesspoint”, which forwards client requests toother servers, such as sending one or more metadataproxy requests to oneof several metadataproxy machines, on behalf of the client or end user.

Venue-Aware Media Application Features

In accordance with an embodiment, the system can include the use ofcrowdsourced microphone or other information with a digital mediacontent environment, so that upon determining that the media device ispresent within a particular venue or attending a particular performance,the media server can enable one or more venue-aware media applicationfeatures, to provide various types of functionalities that improve theuser experience.

FIG. 2 illustrates use of crowdsourced microphone or other informationwith a digital media content environment, in accordance with anembodiment.

As illustrated in FIG. 2, in accordance with an embodiment, a user maybring their media device with media application to a venue/performance240, for example to a theater, or a music concert, or another type ofevent which the user desires to attend 242.

In accordance with an embodiment, a microphone and/or other hardwarecomponents 252 of the media device can be used to indicate a presence254 of the media device within the venue or at the performance.

In accordance with various embodiments, the determination of a mediadevice's presence (and user attendance) at a particular venue or at theperformance can be achieved in a number of ways, such as for example:matching a GPS location of the media device to a time and location ofknown performances; hearing, through the device's microphone, a musicalperformance taking place at the venue; cross-referencing the attendanceof nearby (e.g., via Bluetooth, GPS, or ultrasound) user devices whichhave already registered their attendance; identifying a third-partysocial media posting referencing a user's attendance, such as a Snapchator Instagram message/image that includes a metadata containing locationcoordinates; predicting a user's attendance based on a known ticketpurchase by the user; and/or explicitly registering a user's attendancebased on a direct interaction from the user.

For example, a media application (e.g., a Spotify app) can be configuredto prompt the user with a question “Are you at the (band name)concert?”, and present “Yes”/“No” options to enable the user to settheir particular attendance state.

In accordance with an embodiment, such presence information can befurther determined by a venue/performance system 256 that is associatedwith a particular venue or performance, and can be provided as acombination of venue/performance/presence information 258 to a mediaserver.

Alternatively, in accordance with an embodiment, thevenue/performance/presence information can be communicated directly fromthe media device to the media server.

In accordance with an embodiment, the system can also, or alternatively,utilize an interaction information 262 by the media device (or by theuser thereof) with other systems or environments 264, for example withsocial messaging environments at which the user has indicated theirintent to attend a particular venue or performance.

Such interaction information 266 can then be similarly provided to themedia server, either via those other systems or environments, or asdescribed above directly from the media device to the media server.

In accordance with an embodiment, the media application can beconfigured so that, upon determining that the media device (and the userthereof) is present within a particular venue or attending a particularperformance, the media server can enable 272 one or more venue-awaremedia application features 274.

In accordance with an embodiment, the venue-aware media applicationfeatures can then be used to modify the user environment 276, forexample by modifying the operation or appearance of the user interfaceto include a venue-aware interface 278, or to provide other types offunctionalities that improve the user experience.

For example, a user may desire to attend a concert where a band isperforming. Realizing where it is located, the media application (e.g.,a Spotify app) on the user's media device (e.g., smartphone) can usedevice components such as its illuminated device screen, touch inputinterface, microphone, accelerometer, GPS radio, or other devicecomponents and features, to transform the media device into aninteractive multimedia extension of the live concert experience, for theindividual user and the crowd as a whole, to everyone's surprise anddelight.

In accordance with an embodiment, the features and functionality can becreated and customized in a way that is specific to, e.g., the band orthe particular performance. In crafting such an interactive experience,the designer can draw from a toolbox of enhanced features which havebeen developed beforehand, and are made available to the designer, someexamples of which enhanced features are described in further detailbelow.

FIG. 3 further illustrates use of crowdsourced microphone or otherinformation with a digital media content environment, in accordance withan embodiment.

As illustrated in FIG. 3, in accordance with an embodiment, in somevenues or performances a crowd, group, or other plurality of users 302,having a plurality of media devices 312, may be in attendance, forexample, at a music concert.

For example, as illustrated in FIG. 3, in addition to the original userand their media device, both user 304 and user 306 also have mediadevices 314, 316 respectively, each of which are configured to supportvenue-aware media application features as described above.

In accordance with an embodiment, when the users enter thevenue/performance, their media devices can register its presence 320,and/or the system can also utilize interaction information 322 by thedevice (i.e., user), determine the presence of multiple media devicesand users in attendance. The media server can then enable one or morevenue-aware media application features on each of the media devices 324,to modify the user environments at those devices.

In accordance with an embodiment, in addition to establishing a user'sattendance, the system can continue to use hardware on the mobile deviceto collect additional data, for example to continue to enhance andmaintain the quality of the enhancements to the user experience.

For example, in accordance with an embodiment, the media devicemicrophone can enable synchronization with the music playing. Ratherthan simply relying on an external metadata about the performance thatsays, e.g., song A will start at 10:35 PM, a live recorded waveformreceived via the microphone can be used to enable a precisesynchronization if the band happens to begin the next song a few minuteslate.

As another example, in accordance with an embodiment, the media devicemicrophone can be used to assist the system to stay in more precisesynchronization with the audio the user hears from the stage andspeakers. In a large venue, users can perceive the amount of time thesound takes to travel. Instead of timing all of the media devices to acentralized clock, the waveform captured from the device microphone canmore precisely control the elements of the enhanced user experience.

As another example, in accordance with an embodiment, the microphone canbe used to determine an exact physical location within three-dimensionalspace in the venue. A strong single noise played from two or threesingle sources around the perimeter of the venue will reach devices atdifferent locations within the venue at slightly different times. Usingtriangulation and time-of-flight, the system can determine where withinthe venue a device is currently being held, and update that location asusers move around the venue.

As another example, in accordance with an embodiment, the media devicecamera allows precise synchronization to the lighting, to create astrobe effect, for example by incorporating both environmental lightsand the screens of visiting devices, which is more effective than ifsuch functionality was timed based solely on device clocks and mobilephone network time sources.

In accordance with an embodiment, when the media device (i.e., user)leaves the venue, the enhanced user experience stops, and the mediaapplication resumes its usual operation, for example presenting a“goodbye” interaction to the user, asking a final question, presenting afinal piece of information, or soliciting a last third-party socialmedia posting from the user.

FIG. 4 illustrates a flowchart of a method for use of crowdsourcedmicrophone or other information with a digital media contentenvironment, in accordance with an embodiment.

As illustrated in FIG. 4, in accordance with an embodiment, at step 330,a media application is provided at a media device, for use in accessingmedia content at a media server, for playing at the media device.

At step 332, the media server determines, upon receipt of informationfrom one or more of the media device, media application, avenue/performance system, or other system or environment, that the mediadevice (and user operating the media device) is attending a particularvenue or performance.

At step 334, the operation of the media application (and userenvironment) is modified, including enabling venue-aware mediaapplication features, for at least one or a plurality of media devices(and users operating those media devices) located at the particularvenue/performance, using information received from the media server.

At step 336, a determination is made, upon receipt of information fromone or more of the media device, media application, a venue/performancesystem, or other system or environment, that the media device(s) (anduser(s) operating those media device(s)) are no longer located at theparticular venue/performance, and the usual operation of the mediadevice and media application is restored.

Enhanced Features and Applications

As described above, in accordance with an embodiment, a designer of aninteractive user experience can draw from a toolbox of enhanced featuresdeveloped beforehand, and made available to the designer, some examplesof which are illustrated below:

Lighting: in accordance with an embodiment, venue lighting may beattached to a programmed system to show patterns and effects tocomplement the music while the concert happens. The techniques describedherein enable the illuminated screens of mobile devices in attendance tothe landscape of light-producing devices at the event designer'sdisposal.

Audio playback: in accordance with an embodiment, the speakers on themedia device can produce audio to complement the speakers installed atthe venue. Some instruments, tracks, or effects might play from userdevices instead of installed speakers.

Audio recording: in accordance with an embodiment, mastering a track ofthe live performance, the band's audio engineers can incorporate audiorecorded from the audience not only from microphones installedthroughout the venue, but also from the microphones attached toattendee's devices.

Singing and lyrics: in accordance with an embodiment, the media devicescreen can show lyrics that appear in time while the song plays in thevenue. Users singing along can have their audio incorporated into thelarger experience.

Touch: in accordance with an embodiment, the device touch display couldsuddenly produce audio for the user as part of the experience, either onthe device, surrounding devices, or venue speakers, giving the user amusical instrument, like a drum, to tap on. Alternatively, input fromthe device touch panel could control light in the venue or a surroundingvicinity display.

Attendance: in accordance with an embodiment, display screens in thevenue can list statistics about visitors, such as the number ofattendees who are participating in the enhanced experience. Venuescreens can display notable attendees, while screens of other attendeesdisplay large arrows, pointing them in the direction of the nearestsuperfan. Precise location positioning and six-axis orientation datafrom the accelerometer allows the arrows to keep pointing in the samedirection as users move their devices.

Social media posts: in accordance with an embodiment, when identified atthe venue and part of the experience, the user can be given additionalcapabilities to make special social media postings, includingfunctionality, like a Snapchat or Snapchat-like photography filter, alsoonly available to the user while he or she is at the concert.

Data mining: in accordance with an embodiment, after the event hasconcluded, organizers can access a list of who was in attendancecomplete with their user identity, and third-party social mediaidentities.

The above examples of enhanced features are provided by way of example,to illustrate some of the types of features which the system cansupport. In accordance with various embodiments, additional and/ordifferent enhanced features and applications can be supported.

Embodiments of the present invention may be conveniently implementedusing one or more conventional general purpose or specialized digitalcomputer, computing device, machine, or microprocessor, including one ormore processors, memory and/or computer readable storage mediaprogrammed according to the teachings of the present disclosure.Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art.

In some embodiments, the present invention includes a computer programproduct which is a non-transitory storage medium or computer readablemedium (media) having instructions stored thereon/in which can be usedto program a computer to perform any of the processes of the presentinvention. Examples of the storage medium can include, but is notlimited to, any type of disk including floppy disks, optical discs, DVD,CD-ROMs, microdrive, and magneto-optical disks, ROMs, RAMs, EPROMs,EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards,nanosystems (including molecular memory ICs), or any type of media ordevice suitable for storing instructions and/or data.

The foregoing description of embodiments of the present invention hasbeen provided for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseforms disclosed. Many modifications and variations will be apparent tothe practitioner skilled in the art.

For example, while the techniques described above generally illustrateexamples such as a music streaming service such as Spotify, and streamedmusic or song content, the systems and techniques described herein canbe similarly used with other types of media content environments, andother types of streamed data or media content.

The embodiments were chosen and described in order to best explain theprinciples of the invention and its practical application, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with various modifications that are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. A system for use of crowdsourced microphone orother information with a digital media content environment, comprising:a media device including a media application, and a microphone or otherhardware components of the media device that can be used to indicate apresence of the media device to a media server; and wherein the mediaapplication can be configured so that, upon determining that the mediadevice is one or more of present within a particular venue or attendinga particular performance, the media server can enable one or morevenue-aware media application features.
 2. The system of claim 1,wherein presence information is determined by one or more venue orperformance system that is associated with a particular venue orperformance, and is provided as a combination of one or more venue,performance, or presence information, to a media server.
 3. The systemof claim 1, wherein the one or more venue, performance, or presenceinformation is communicated directly from the media device to the mediaserver.
 4. The system of claim 1, wherein the venue-aware mediaapplication features can be used to modify the user environment,including modifying the operation or appearance of the user interface toinclude a venue-aware interface.
 5. A method for use of crowdsourcedmicrophone or other information with a digital media contentenvironment, comprising: providing, at a media device having amicrophone or other hardware components of the media device that can beused to indicate a presence of the media device to a media server, amedia application; and wherein the media application can be configuredso that, upon determining that the media device is one or more ofpresent within a particular venue or attending a particular performance,the media server can enable one or more venue-aware media applicationfeatures.
 6. The method of claim 5, wherein presence information isdetermined by one or more venue or performance system that is associatedwith a particular venue or performance, and is provided as a combinationof one or more venue, performance, or presence information, to a mediaserver.
 7. The method of claim 5, wherein the one or more venue,performance, or presence information is communicated directly from themedia device to the media server.
 8. The method of claim 5, wherein thevenue-aware media application features can be used to modify the userenvironment, including modifying the operation or appearance of the userinterface to include a venue-aware interface.
 9. A non-transitorycomputer readable storage medium, including instructions stored thereonwhich when read and executed by one or more computers cause the one ormore computers to perform the steps comprising: providing, at a mediadevice having a microphone or other hardware components of the mediadevice that can be used to indicate a presence of the media device to amedia server, a media application; and wherein the media application canbe configured so that, upon determining that the media device is one ormore of present within a particular venue or attending a particularperformance, the media server can enable one or more venue-aware mediaapplication features.
 10. The non-transitory computer readable storagemedium of claim 9, wherein presence information is determined by one ormore venue or performance system that is associated with a particularvenue or performance, and is provided as a combination of one or morevenue, performance, or presence information, to a media server.
 11. Thenon-transitory computer readable storage medium of claim 9, wherein theone or more venue, performance, or presence information is communicateddirectly from the media device to the media server.
 12. Thenon-transitory computer readable storage medium of claim 9, wherein thevenue-aware media application features can be used to modify the userenvironment, including modifying the operation or appearance of the userinterface to include a venue-aware interface.